The Inclusive Classroom Profile (ICP) 

Preliminary Findings of Demonstration Study in North Carolina 


Abstract 

The Inclusive Classroom Profile (ICP) was developed in response to a lack of validated instruments 
designed specifically to measure the quality of inclusive practices, and it is based on research 
evidence on the effectiveness of specialized instructional strategies for meeting the individual needs 
of children in inclusive settings (Odom, 2004; National Professional Development Center on 
Inclusion, 2011; Buysse and Hollingsworth, 2009). The ICP has been used in a pilot study in the 
United Kingdom with promising results for the measure’s reliability and validity (Soukakou, in 
press). The purpose of this study was to conduct the first USA demonstration study leading to 
validation. The measure was field tested in 51 inclusive settings in North Carolina. The following are 
the preliminary results from this demonstration study: 

• The measure has acceptable inter-rater agreement, is internally consistent, and shows a good 
factor structure. 

• Correlations with another measure of global classroom quality (ECERS-R) provided initial 
evidence for construct validity. 

• Assessors reported the measure to be feasible and easy to use. 

Background/rationale 

Never has so much attention been paid to improving the quality of early childhood programs serving 
high- need children. Through the Race to the Top -Early Eearning Challenge Program and other 
reform efforts, states are being challenged to develop systems for rating, monitoring and improving 
early learning and development programs. States need reliable, valid tools for measuring classroom 
quality that are sensitive to and inclusive of each and every child. A pilot study, conducted by PPG in 
collaboration with the NC Rated Eicense Assessment Project; the Division of Child Development 
and Early Education; and the NC Department of Public Instruction, Exceptional Children 
investigated a new measure, the Inclusive Classroom Profile (ICP; Soukakou 2012). The ICP is a 
structured observation rating scale designed to complement existing program quality measures with 
new quality indicators that assess specific, classroom practices that support the developmental 
progress of children with disabilities in early childhood settings. The ICP was developed in response 
to a lack of validated instruments designed specifically to measure the quality of inclusive practices, 
and it is based on research evidence on the effectiveness of specialized instructional strategies for 
meeting the individual needs of children in inclusive settings (Odom, 2004; National Professional 
Development Center on Inclusion, 2011; Buysse and Hollingsworth, 2009). The ICP has been used in 
a pilot study in the United Kingdom with promising results for the measure’s reliability and validity 
(Soukakou, in press). This is the first US demonstration study leading to validation. 

Research Objectives 

The study involved field testing the ICP measure in 51 settings in NC with the following objectives: 

(1) Examine the acceptability of the ICP measure for program evaluation within an accountability 
structure such as quality rating systems. 

(2) Assess the effectiveness of training procedures for ensuring accurate administration. 

(3) Assess the psychometric properties of the ICP, including the measure’s inter-rater reliability, 
factor structure and construct validity. 

(4) Explore the relationship between inclusive quality in NC settings and various child and 
classroom characteristics (e.g. relationship between quality and teachers’ special education 
qualifications, number of children with special needs and severity of disability). 
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Sample 

The ICP was fieldtested in 51 inclusive preschool classrooms. Criteria for participation in the study 
included: 1) The program was center-based 

2) The program served preschool age children (three-five years of age) 

3) The program served at least one child and no more than 50% with an lEP. 

Classroom characteristics 

Of the 51 settings, 39% were child care programs, 26 % were Developmental Day programs, 26% 
were Head Start programs and 9% were public preschools. In total, classrooms included 150 
children with an identified disability (child with an lEP). The number of children with a disability per 
classroom ranged from one to eight children, with each classroom serving in average 2.94 children 
with an lEP. Teacher education level ranged from a high school diploma through a Master’s degree, 
with 51% of the teachers reporting having received a Bachelor’s degree. 

Area and severity of disability 

Eor each child with a disability (A^=150) teachers were asked to identify children’s area and severity 
of disability'. Children’s special needs were identified in the following areas: behavior/social 
(67%); gross motor (27%); fine motor coordination (45%); sensory integration (27%), and intentional 
communication (90%). With regard to severity of disability, 59% of the classrooms had a least one 
child with a disability identified at the ‘severe’ level (1-4 point scale rating 
possible/mild/moderate/severe level), while 88% of classrooms had at least one child with a 
moderate or severe level of disability. 


Data Collection 

Eour assessors from the North Carolina Rated Eicense Assessment Project (NCREAP) collected data 
over a four month period using the following measures: Inclusive Classroom Profile (ICP; 
Soukakou, 2007), and the Early Childhood Rating Scales (ECERS-R; Harms, Clifford and Cryer, 
2005). The NCREAP has the contract in NC to collect data, including ECERS-R for NC QRIS. Eor 
this study, the assessors were formally trained in using the ICP. Information on children and 
classrooms was gathered through teacher interview and was recorded on a classroom information 
form developed by the researchers of the study. Information on the acceptability and usability of the 
ICP was gathered through a social validity survey completed by the assessors. Assessors also 
participated in a full-day focus-group to provide feedback on the measure and its demonstration. 


Key Findings 

Objective 1: Examine the acceptability of the ICP measure for program evaluation. 

Results from the social validity survey (1-5 point scale), indicated that assessors rated the importance 
of the constructs measured by the ICP very highly (m= 5) and that they would highly recommend the 
ICP measure to others (m=5). With regard to the usability of the ICP, the assessors found the 
measure easy to use (m= 4). Einally, assessors rated the ‘Overview’ part of the training (information 
on administration and scoring, including video demonstration) as useful (m= 3.75), and all four 
assessors reported that they felt well prepared after the reliability training observations (m=4). 
Eindings from the focus-group feedback meeting confirmed the high level of acceptability of the ICP 


' Teachers could identify one or more areas of need for each child with an lEP. 
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measure by the assessors who used the ICP in the study, and revealed specific recommendations for 
improving the administration, scoring and training procedures. 

Objective 2: Examine the effectiveness of training procedures for accurate and reliable 
administration. 

Assessors received a three -hour training session on ICP administration and scoring followed by 4 
reliability observations in inclusive classrooms to assess accuracy of administration and reliability 
proficiency against the ICP author standard. Each assessor (n=4) met a reliability-proficiency 
criterion of 85% agreement within one scale point, maintained for three consecutive reliability 
observations against the ICP author’s rating. Mean inter-rater agreement across assessors was 98% 
with a range of 91-100%. 

Objective 3: Assess the psychometric properties of the ICP, including the measure’s inter-rater 
reliability, factor structure and construct validity. 

Inter-rater reliability was assessed in 9 reliability observations (18% of the sample), distributed over 
the four month data collection period. The mean inter-rater agreement across reliability observations 
was 87% (within one point scale). Reliability at the item level assessed via an intra-class correlation 
coefficient (ICC) ranging from zero to one where one indicates perfect agreement. For most items 
agreement between raters was in an acceptable range (.51 -.99) with a mean of .71. Only one item 
(Item 3) presented with low agreement (.11) indicating the need for further clarification. 

Accuracy of ICP administration by assessors against author standards was assessed in 9 classroom 
observations. The mean agreement between assessor’s scores and ICP author standards (consensus 
score) was 94% (within one) and 75% (exact agreement) indicating that assessors maintained 
accurate administration of the ICP throughout the study. 

Internal consistency Cronbach’s alpha analysis for the 12 items suggested that the scale’s items were 
internally consistent (a = .85). 

Structural validity was assessed through exploratory factor analysis of the ICP items. A scree plot 
examination (see figure 1) of the variance explained by the number of factors, clearly showed a one- 
factor structure with good factor loadings (.34 - .84). 

Construct validity was examined through correlation analysis between the ratings on the ICP in and 
ratings on the Early Childhood Environment Rating Scales (ECERS-R; Harms, Clifford and Cryer, 
2005). A moderate high correlation was found between the total score of the ICP and the ECERS-R 
{rho =.51) suggesting that the two instruments are not measuring identical constructs. A similar 
pattern of weaker correlations between the ICP and certain ECERS subscales, such as the ‘health and 
personal care routines’ that involve practices not measured by the ICP {rho =21) and higher 
correlations with ECERS subscales that measure constructs more closely related to the ICP ( 
‘interactions’; rho = .43) provide preliminary evidence for the construct validity of the ICP. 

Objective 4: Examine the relationship between the quality of observed inclusive practices and 
child and classroom characteristics. 

Multiple regression analyses are being conducted to explore the association between observed quality 
of inclusive practices (as measured by the ICP) and children and classroom characteristics (e.g., 
number of teacher completed course hours in special education, and type and severity of children’s 
disabilities). 
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Significance of findings 

This demonstration study provided further evidence for the reliability, validity, and usability of the 
ICP. Implementation of the ICP in NC classrooms contributed to building an evidence base for this 
instrument that can enable a more comprehensive assessment of the quality of specific classroom 
practices implemented in inclusive programs. Findings from the present study have implications for 
use of the ICP in research, quality rating systems and professional development. 

• Asa research instrument, the ICP could be used to assess quality across inclusive programs, 
as well as to evaluate the relationship between quality of inclusive practices and outcomes 
for children with disabilities and their families. 

• As one component of a quality ratings system, the ICP could be used to evaluate the quality 
of inclusive practices. 

• Asa quality improvement tool, the information gathered on ICP could inform professional 
development models and enhance program quality improvement activities. 

Next Steps: 

1. Revise the ICP rating scale and administration guidelines. 

2. Write an article on the study findings for a peer reviewed journal. 

3. Develop a training system, including online, web-based materials for training assessors to 
reliably administer the ICP. 

4. Develop materials that support the use of the ICP in a professional development context. 

5. Identify and secure funding for items #3 and #4. 
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